Part 13.3 - Machine Learning Python Programming Training

Convolutional Neural Network (CNN)

• Image-specific Artificial Neural Network – image search/recognition, face recognition, face verification, self-driving and more

• Its concept comes from brain visual cortex where many neurons have a small region of the visual field (in CNN, filter)

• The name of CNN comes from “convolution”, one of the most important operation in CNN.

• The first Convolutional Neural Network is LeNet-5, which can classify digits from hand-written number.

• Advantage compared to DNN

  • 2 or 3 dimensional read (image rather than vector)
  • Less parameters, much less computing power

Image Data

When the computer see the image, it sees pixel values.

CNN Architecture

1. Input layer (4 dimensional vector)
2. Convolution layer– Finding local pattern
3. Pooling layer
4. Flatten
5. Fully-connected layers – Finding global pattern
6. Output layer

1. Input layer (4 dimensional vector)

• Import image (.jpg) to numeric number - (28,28) for black/white or (28,28,3) for color
• If needed, convert image data into 3 dimensional vectors : (28,28) to (28,28,1)
• Convert to 4 dimension vector by adding number of images - 60,000 images to 60,000 numeric vector of (60000, 28,28,1)

2. Convolutions

Convolution – the mathematical combination of two functions to produce a third function.

• Filters (Kernel Matrix) are image feature identifiers ( e.g., straight edges and curves)
• Input Image * Kernel
  • If high number, input data has filter(kernel) image.
  • If low number, input data does NOT have filter(kernel) image.

3. Pooling

Pooling with filters of 2 X 2

• Max Pooling – more popular than average pooling

• Average Pooling : (-5 -4 -10 -2) / 4 = -21/4= -5.25 Pooling progressively reduces the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting.

4. Flatten Layer - Architecture

The flattening section of CNN architecture will flatten a pooled layer to one column so that the data will be passed through ANN to get processed further. The neuron of flatten layer detects a certain feature like nose, mouth,,,
In this CNN architecture, pooling layer of (4,4,20) will be flattened to flatten layer of (4x4x20=320), which means that it has 320 image features.

5. Fully-connected layer

The fully-connected layer section of CNN has three layers.

• Flatten layer
• Fully-connected layer
• Output layer Fully-connected layers in CNN works as follows
• The neurons in fully-connected layer detect a certain feature of the image like nose, mouth or eyes
• It combines the features into a wider variety of attributes that makes CNN more capable for classifying the images

6. Output Layer

The output layer section outputs an N (e.g., 10) dimensional vector where N is the number of classes that the program has to choose from.

For example,

• The resulting vector for a digit classification program is [0,.1, .1, 0, 0, 0.75, 0, 0, 0, .05] for number 0 to 9
• It represents a 10% probability that the image is “1”, a 10% probability that the image is “2”, a 75% probability that the image is “5”, and a 5% probability that the image is “9”.
• Model will predict the image is “5”.

How Image features are progressed throughout CNN architecture

Famous CNN Architecture competition winner

• AlexNet (developed in 2012)
  • Famous winner of the 2012 ImageNet LSVRC-2012 competition by a large margin (15.3% VS 26.2% (second place) error rates)
• ZFNet (in 2013)
  • Top 5 error rate: 14.8%
• GoogLeNet(19) - 2014
  • Top 5 error rate: 6.67%
  • Number of Parameters : 4 million
• VGG Net(16) - 2014
  • Top 5 error rate: 6.67%
  • Number of Parameters : 4 million
• ResNet(152) - 2015
  • Top 5 error rate: 3.6%

Convolution Python Programm

## Keras for DNN
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras.callbacks import EarlyStopping
from keras.optimizers import Adam
import numpy as np

Below is CNN acchitecture that the data scientist builds for the workshop.

• Input layer : 28 by 28 pixel images
• First Convolution : 64 (3 by 3 ) filters
• First Pooling : (2 by 2 ) Maximum value pooling
• Second Convolution : 128 (3 by 3 ) filters
• Second Pooling : (2 by 2 ) Maximum value pooling
• Third Convolution : 256 (3 by 3 ) filters
• Third Pooling : (2 by 2 ) Maximum value pooling
• First Dense Layer : 128 neurons
• Output Layer : binary ( 0 or 1)

## Buiid CNN model

model_cnn = Sequential()

## First Convolution and Pooling
model_cnn.add(Conv2D(64, kernel_size=(3, 3), activation='relu', input_shape=(28,28,1)))
model_cnn.add(MaxPooling2D(pool_size=(2, 2)))

## Second Convolution and Pooling
model_cnn.add(Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same'))
model_cnn.add(MaxPooling2D(pool_size=(2, 2)))

## Third Convolution and Pooling
model_cnn.add(Conv2D(256, kernel_size=(3, 3), activation='relu', padding='same'))
model_cnn.add(MaxPooling2D(pool_size=(2, 2)))

## Flatten Layer
model_cnn.add(Flatten())

## Fully-Connected Layer
model_cnn.add(Dense(128, activation='relu'))
model_cnn.add(Dropout(0.5))

## Output Layer
model_cnn.add(Dense(2, activation='softmax'))

## Compile CNN model
model_cnn.compile(loss=['categorical_crossentropy'], optimizer='adam', metrics=['accuracy'])

model_cnn.summary()

Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_3 (Conv2D)            (None, 26, 26, 64)        640       
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 13, 13, 64)        0         
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 13, 13, 128)       73856     
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 6, 6, 128)         0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 6, 6, 256)         295168    
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 3, 3, 256)         0         
_________________________________________________________________
flatten_3 (Flatten)          (None, 2304)              0         
_________________________________________________________________
dense_12 (Dense)             (None, 128)               295040    
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_13 (Dense)             (None, 2)                 258       
=================================================================
Total params: 664,962
Trainable params: 664,962
Non-trainable params: 0
_________________________________________________________________

## Train the model model_cnn.fit(X_tain, y_train, batch_size=128, epoch=10, validation_split=0.2) ## Predict with the model cnn_preds = model_cnn.predict(X_test) ## Evaluate the trained model model_cnn.evaluate(X_test, y_test)

CNN Implementation

• One of the most popular DNN
• Go-to-model for all the image-related problems
• Image Classification
• X-ray diagnosis
• Face Recognition
• Face Verification
• Object detection
• Autonomous vehicles

Recurrent Neural Network

Introduction – recurrent neural network model to use sequential information.

Why RNN?

• In traditional ANN, all inputs and outputs are independent of each other. But, in some case, they could be dependent.
• RNN is useful when inputs are dependent.
• Some problems such as text analysis and translation, we need to understand which words come before.
• RNN has a memory which captures previous information about what has been calculated so far.
  

Basic RNN Structure and Algorithms

RNN unit - LSTM (Long Short-Term Memory Unit)

• It is composed of 4 gates – input, forget, gate and output.
• LSTM remembers values over arbitrary time intervals and the 3 gates regulate the flow of information into and out of LSTM unit.
• LSTMs were developed to deal with the vanishing gradient problems.
• Relative insensitivity to gap length is an advantage of LSTM over RNNs.

Popular RNN Architecture

Text notation. # of inputs = # of outputs

Sentimental Analysis (PV signal) : x = text, y = 0/1 or 1 to 5

Music generation \ Picture Description: x= vector, y = text

Machine translation : x = text in English, y = text in French

Natural Language Processing (NLP) using RNN

Introduction of NLP – An area of artificial intelligence on an interaction between computer and human natural language. NLP can program computers to process and analyze natural language data.

• Input data – Language
• Output data - Language

Simple RNN architecture using NLP

• Input data – “I am smiling”, “I laugh now”, “I am crying”, “I feel good”, “I am not sure now”
• Embedding – to convert words to vector number
• LSTM – to learn language
• Softmax – to provide probability of output
• Output data - “very unhappy”, “unhappy”, “happy”, “very happy”

Natural Language Processing (NLP) procedures

1. Import data and preparation
2. Tokenizing – representing each word to numeric integer number : “the” to 50
3. Padding – fixing all the records to the same length
4. Embedding – representing word(numeric number) to vectors of numbers
   5o to [ 0.418, 0.24968, -0.41242, 0.1217, 0.34527, -0.044457, -0.49688, -0.17862, -0.00066023,,,,, ]
5. Training with RNN models

1. Import Data and Preparation

• Import document to working area
• Convert document to sentences
• Split sentence to words
• Filter out punctuation
• Convert words to lower case

## Python Programming for importing word preparation
from keras.preprocessing.text import text_to_word_sequence

texts = ['I do not like her.', 'I love her']
line =[ ]
for text in texts:
    words = text_to_word_sequence(text)
    line.append(words)
print(line)

labels = [1,0]

[['i', 'do', 'not', 'like', 'her'], ['i', 'love', 'her']]

2. Tokenization

• Tokens - words
• Tokenizing - Representing word(numeric number) to integer number
• Word “the” to 1
• Words are represented by dense vectors where a vector represents the projection of the word into a continuous vector space

## Python Programming for importing word preparation
from keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer()
tokenizer.fit_on_texts(line)
tokenized_words = tokenizer.texts_to_sequences(line)
print(tokenized_words)

[[1, 3, 4, 5, 2], [1, 6, 2]]

3. Padding

• Padding – Preparing tokenized input data into the same length so that input data can go into Embedding layer
• It will make all the records to fixed, consistent length.

from keras.preprocessing.sequence import pad_sequences
padded_lines = pad_sequences(tokenized_words , maxlen=5, padding='post')
print(padded_lines )

[[1 3 4 5 2]
 [1 6 2 0 0]]

4. Embedding

• Representing word(numeric number) to vectors of numbers
  Word “the” to [ 0.418, 0.24968, -0.41242, 0.1217, 0.34527, -0.044457, -0.49688, -0.17862, -0.00066023,,,,, ]
• Words are represented by dense vectors where a vector represents the projection of the word into a continuous vector space
• Embedding layer has embedding matrix which will be trained as a part of model.
• The shape of embedding matrix is (vocabulary size, # of vector )
• Embedding layer types
• Keras Embedding layer – Random initiation on embedding matrix
• Pre-trained embedding methods
  • Word2Vec
  • GloVe

## import gensim model from gensim.models import Word2Vec## creat NLP model using gensim for embeding model_NLP = Word2Vec(line, min_count=1) print(model_NLP.wv.vocab) print(line) print(model_NLP['i']) print(model_NLP['i'].shape)

5. Train with RNN model

## import RNN model
from keras.preprocessing.text import Tokenizer
from keras.models import Sequential
from keras.layers import Dense, LSTM, Embedding

tokenizer = Tokenizer() tokenizer.fit_on_texts(X_train_rnn) X_input = tokenizer.texts_to_sequence(X_train_rnn)

model_rnn = Sequential()
model_rnn.add(Embedding(input_dim=40, output_dim=100, input_length=5))
model_rnn.add(LSTM(10))
model_rnn.add(Dense(10, activation='relu'))
model_rnn.add(Dense(1, activation='softmax'))

model_rnn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

model_rnn.summary()

Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_2 (Embedding)      (None, 5, 100)            4000      
_________________________________________________________________
lstm_2 (LSTM)                (None, 10)                4440      
_________________________________________________________________
dense_14 (Dense)             (None, 10)                110       
_________________________________________________________________
dense_15 (Dense)             (None, 1)                 11        
=================================================================
Total params: 8,561
Trainable params: 8,561
Non-trainable params: 0
_________________________________________________________________

## Final Data Preparation
X_train_rnn = padded_lines
y_train_rnn = np.array(labels) ## update

print("X training shape : ", X_train_rnn.shape)
print("y training shape : ", y_train_rnn.shape)

X training shape :  (2, 5)
y training shape :  (2,)

## Train RNN model with 
model_rnn.fit(X_train_rnn, y_train_rnn, epochs=10)

Epoch 1/10
1/1 [==============================] - 1s 1s/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 2/10
1/1 [==============================] - 0s 3ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 3/10
1/1 [==============================] - 0s 4ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 4/10
1/1 [==============================] - 0s 2ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 5/10
1/1 [==============================] - 0s 3ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 6/10
1/1 [==============================] - 0s 2ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 7/10
1/1 [==============================] - 0s 2ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 8/10
1/1 [==============================] - 0s 3ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 9/10
1/1 [==============================] - 0s 2ms/step - loss: 0.0000e+00 - accuracy: 0.5000
Epoch 10/10
1/1 [==============================] - 0s 2ms/step - loss: 0.0000e+00 - accuracy: 0.5000

<keras.callbacks.History at 0x2888f23bd60>

RNN Implementation

• Speech Recognition
• Music Generation
• Sentiment Classification
• Machine Translation
• Video Activity Recognition
• Texts Classification and Extraction
• DNA Sequence Analysis
• Chatbot

Who will learn faster Python programming

• SAS Programmers
• Novice Programmers

Why experience programmers can learn other or new language faster?

Transfer Learning

Machine Learning method where a pre-trained model is reused as a starting point of the model development for another/similar task.

Popular pre-trained model

For Image Data

• Xception
• ImageNet
• ResNet
• Inception
• Oxford VGG model
• Google Inception model
• Microsoft ResNet Model

For Language Data

• Google’s word2vec model
• Stanford’s GloVe model
• ULMFit from fast.ai
• BERT
• ElMO

Keras Transfer Learning

• Xception
• VGG16, VGG19
• ResNet, ResNetV2, ResNet
• InceptionV3, InceptionResNetV2
• MobileNet, MobileNetV2

How model learns in Image Recognition

• Early layers – Edges Detection
• Middle layers – Shapes Detection
• Later layers – High Level Features Detection

Pre-trained Model : VGG

• Architecture – 16 layers or 19 layers
• CNN model
• 14,000,000 images with 1000 class objects
• 138,000,000 parameters
• With NVIDIA Titan Black GPU, training takes a week.
• Available in public

Transfer Learning Implementation

1. Select pre-trained model
2. Import pre-trained model
3. Determine which parameters/layers of the pre-trained model should be trained
4. Add more layers of the imported pre-trained model
5. Set up the output layers
6. Prepare data features (input data) for pre-trained model
7. Compile pre-trained and added layers
8. Train the new model

Why Transfer Learning?

• Training Faster
• A lack of data
• Cost-effective. It is expensive to train the complex CNN/RNN model. The most complex CNN/RNN model will take weeks/months to train using hundreds of GPU/TPU equipped computers.

## import pre-trained model
from keras.applications import vgg16
from keras.layers.core import Dense, Flatten
from keras.layers import Input
from keras.models import Model, Sequential

## import VGG16 model for transfer learning
input_shape= (224, 224, 3)
vgg_model = vgg16.VGG16(include_top=False, weights='imagenet', input_shape=input_shape)

VGG16 model in Keras

vgg16_model = Sequential()

vgg16_model.add(Conv2D(64, (3, 3), input_shape=input_shape, padding='same', activation='relu'))
vgg16_model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
vgg16_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

vgg16_model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
vgg16_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

vgg16_model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
vgg16_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

vgg16_model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
vgg16_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

vgg16_model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
vgg16_model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
vgg16_model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

vgg16_model.add(Flatten())
vgg16_model.add(Dense(4096, activation='relu'))
vgg16_model.add(Dense(4096, activation='relu'))
vgg16_model.add(Dense(1000, activation='softmax'))

## Summary of pre-trained VGG model
vgg_model.summary()

Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0
_________________________________________________________________

#Create your own input format (here 3x200x200)
vgg_input = Input(shape=(224, 224, 3),name = 'image_input')

## add custom layers to pre-trained VGG model
X = vgg_model.output
X1 = Flatten()(X)
X2 = Dense(1024, activation='relu')(X1)
X3 = Dense(1024, activation='relu')(X2)
Target = Dense(15, activation='softmax')(X3)

model_vgg_tf = Model(vgg_model.input, Target)

model_vgg_tf.compile(loss="categorical_crossentropy", optimizer='adam', metrics=['accuracy'])
model_vgg_tf.summary()

Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0         
_________________________________________________________________
flatten_4 (Flatten)          (None, 25088)             0         
_________________________________________________________________
dense_16 (Dense)             (None, 1024)              25691136  
_________________________________________________________________
dense_17 (Dense)             (None, 1024)              1049600   
_________________________________________________________________
dense_18 (Dense)             (None, 15)                15375     
=================================================================
Total params: 41,470,799
Trainable params: 41,470,799
Non-trainable params: 0
_________________________________________________________________

## Train the model model_vgg_tf.fit(X_tain_tf, y_train_tf, batch_size=128, epoch=10, validation_split=0.2) ## Predict with the model vgg_tf_preds = model_vgg_tf.predict(X_test_tf) ## Evaluate the trained model model_vgg_tf.evaluate(X_test_tf, y_test_tf)

## add custom layers to pre-trained VGG model
## Build the same model with the different method
model_vgg_tf2 = Sequential()
model_vgg_tf2.add(vgg_model)
model_vgg_tf2.add(Flatten())
model_vgg_tf2.add(Dense(1024, activation='relu'))
model_vgg_tf2.add(Dense(1024, activation='relu'))
model_vgg_tf2.add(Dense(15, activation='softmax'))

model_vgg_tf2.compile(loss=['categorical_crossentropy'], optimizer='adam', metrics=['accuracy'])
model_vgg_tf2.summary()

Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
vgg16 (Functional)           (None, 7, 7, 512)         14714688  
_________________________________________________________________
flatten_5 (Flatten)          (None, 25088)             0         
_________________________________________________________________
dense_19 (Dense)             (None, 1024)              25691136  
_________________________________________________________________
dense_20 (Dense)             (None, 1024)              1049600   
_________________________________________________________________
dense_21 (Dense)             (None, 15)                15375     
=================================================================
Total params: 41,470,799
Trainable params: 41,470,799
Non-trainable params: 0
_________________________________________________________________

BERT Model

• Bidirectional Encoder Representation from Transformers
• The first deeply bi-directional, unsupervised language representation.
• State of the art language model for 104 languages.

BERT Training

BERT uses two training strategies

1. Masked ML - Before feeding into BERT, 15% of the words are replaced with mask(empty) token and BERT model is trained

1. Next Sentence Prediction
   • Input – pairs of sentences
   • Three embedding – token, sentence and positional
   • Labels - IsNext / NotNext

Use pre-trained Mask BERT model to fill the word in blank

Predict the word in a situation that it is missing.
"The first death in the US [blank] reported on February 29"
What could be included in here?

Import pre-trained BERT Library

## import BERT model
import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

Create BERT Tokenizer and feature preparation

# Load pre-trained model tokenizer (vocabulary)
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenized input
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
text = "The first death in the US was reported on February 29."
bert_tokenized_text = bert_tokenizer.tokenize(text)
print(bert_tokenized_text)

['the', 'first', 'death', 'in', 'the', 'us', 'was', 'reported', 'on', 'february', '29', '.']

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 6
bert_tokenized_text[masked_index] = '[MASK]'
#assert bert_tokenized_text == ['who', 'was', 'jim', 'henson', '?', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer']
## printing
print(bert_tokenized_text)

['the', 'first', 'death', 'in', 'the', 'us', '[MASK]', 'reported', 'on', 'february', '29', '.']

# Convert token to vocabulary indices
bert_indexed_tokens = bert_tokenizer.convert_tokens_to_ids(bert_tokenized_text)
# Define sentence A and B indices associated to 1st and 2nd sentences (see paper)
#segments_ids = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

# Convert inputs to PyTorch tensors
bert_tokens_tensor = torch.tensor([bert_indexed_tokens])
#segments_tensors = torch.tensor([segments_ids])
print(bert_tokens_tensor)

tensor([[1996, 2034, 2331, 1999, 1996, 2149,  103, 2988, 2006, 2337, 2756, 1012]])

Load pre-trained model

# Load pre-trained model (weights)
model_bertBML = BertForMaskedLM.from_pretrained('bert-base-uncased')

Predict with Pre-trained BERT model

# Predict all tokens
predictions = model_bertBML(bert_tokens_tensor)

# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = bert_tokenizer.convert_ids_to_tokens([predicted_index])
print(predicted_token)

['.']

Use pre-trained BERT model (next sentence prediction)

Predict where the second sentence is the next sentence.

Imporrt BERT Next Sentence Prediction library

import torch
from pytorch_pretrained_bert import BertTokenizer, BertForNextSentencePrediction


# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')

Prpeare input data (features)

sentA = "[CLS]" + " What Is and Isn’t Affected by the Government Shutdown " + "[SEP]"
sentB = " Transportation Security Administration officers checking passengers at Pittsburgh International Airport last week. The agency’s employees have called out sick in increased numbers across the country since the shutdown began. " +"[SEP]"

# sentA = "[CLS]" + " Meanwhile: For a Knife, Dagger, Sword, Machete or Zombie-Killer, Just Ask These Ladies "+ "[SEP]"
# sentB = " Whitehead’s Cutlery in Butte, Mont., is 128 years old and will gladly sharpen scissors sold generations ago. "+"[SEP]"

text = sentA + sentB

tokenized_sentA = tokenizer.tokenize(sentA)
tokenized_sentB = tokenizer.tokenize(sentB)

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_sentA + tokenized_sentB)
segments_ids = [0] * len(tokenized_sentA) + [1] * len(tokenized_sentB)

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

Import pre-trained BERT model for next sentence prediction

# Load pre-trained model (weights)
model = BertForNextSentencePrediction.from_pretrained('bert-large-uncased')

Predict if it is the next sentence

# Predict is Next Sentence ?
preds = model(tokens_tensor, segments_tensors)    

Next_sentence = torch.nn.functional.softmax(preds, dim=1)[:, 0]
Next_sentence # if it is 1, the second sentence is the next sentence

tensor([1.0000], grad_fn=<SelectBackward>)

CONCLUSION

• Machine Learning Introduction
• How Machine Learning works
• Machine Learning Types and algorithms
• Deep Learning Introduction - CNN, RNN and Transfer Learning

CONTACT INFORMATION

Your comments and questions are valued and encouraged. Please contact at:
Kevin Lee
AVP of Machine Learning and AI Consultant
Kevin.lee@genpact.com
Tweet: @HelloKevinLee
LinkedIn: www.linkedin.com/in/HelloKevinLee/

Questions